One partition's failure no longer ends the maintenance walk (#737) - #747
Conversation
The maintenance walk goes neediest-first (LLP 0199#neediest-first), so the most fragmented partition is visited first, and that is exactly the partition most likely to fail a rewrite. There was no per-partition catch and `withSpan` rethrows, so one torn data file aborted the entire walk: every partition behind it got neither compaction nor snapshot expiry, and the daemon logged `daemon.maintenance_failed` and hit the same wall an hour later. Catch around the `maintenance.partition` span - outside `withSpan`, so the helper still records the exception and an ERROR status on the partition span - record the failure on that partition's report, and continue the walk. The partition report is built before the span opens so the catch can keep what the run had established (live file count, snapshots already expired). LLP 0217's writer-generation stamp is untouched: it is written from inside `compactGeneration`'s own catch, before the error reaches this one, so a failing partition still spends its retry once per writer generation. The tick resolves rather than rejecting, because the walk completed, and carries the failures instead: `failed` / `errorKind` / `errorMessage` per partition and `totalFailed` on the report. Rejecting after the walk would throw away the report, so a tick that maintained forty partitions and lost one would print nothing about any of them. Callers read the failures off the report: the daemon logs one `daemon.maintenance_failed` per failed partition (now naming the dataset and partition) and marks the tick span degraded, and `hyp query maintain` prints a FAILED line and exits non-zero. `failed` is deliberately distinct from LLP 0218's `compactionAttemptFailed`: the first means this tick threw here, the second means an earlier tick's attempt failed and nothing has been attempted since. Settled in LLP 0220. Tests that asserted the whole tick rejected now assert the partition's report; what they pin is unchanged.
neutral review — round 1Head reviewed: VERDICT: findings — 1 major, 3 minor. The core change holds up. The catch is genuinely outside 1. major —
|
- maintenance.js: gate the budget-break guard on partitions actually maintained, not on report count, so a failing first partition cannot stand in for the "one partition maintained" guarantee and starve the walk behind it. Add MAX_FAILURES_BEFORE_BUDGET_BREAK as the bounded escape hatch for an all-failing cache. - maintenance.js: set r.rebaselined only after writeCursor persists it, so a failed rebaseline write is not reported or counted as one. - query.js: write a stderr line on a degraded `hyp query maintain` tick, so a caller that only captures stderr still learns the walk lost a partition. - daemon/runtime.js: manage the maintenance.tick span directly instead of through withSpan, so the tick's real verdict sets the span status code rather than being clobbered by withSpan's pre-callback status snapshot. - llp/0220: correct the consequences wording to the guarantee that actually holds, and describe the span status fix. - Add regression tests for all four fixes, each confirmed to fail against the pre-fix code and pass against the fix. Co-Authored-By: Claude <noreply@anthropic.com>
|
VERDICT: findings neutral review - round 2 (final)Head reviewed: VERDICT: findings - 0 blockers, 0 major, 1 minor, 4 nits. Nothing here is ship-blocking. All four round-1 findings are genuinely fixed, and I could not reproduce any of them at this head. The one minor and the nits are precision/robustness items that a follow-up can absorb. Priority 1 was the 1. minor (not blocking) -
|
neutral review record - round 1 (marker correction)This comment exists to carry the round marker in the form the reconciler counts. Head reviewed:
Full evidence, reproductions and suggested diffs are in the linked comment. |
neutral review record - round 2, final (marker correction)Same correction as the round-1 record above: the full round-2 review is Head reviewed:
All four round-1 findings were re-derived independently and are genuinely fixed; Residual, all follow-up material:
Review budget is now exhausted at 2 rounds. Triage decides ship-or-block next. |
neutral triage - shipReview budget exhausted at 2 rounds with six residual findings open, so triage Deferred to #753, which also picks up a seventh, pre-existing item. Two of the six were checked rather than inherited, because a mis-labelled nit is
On the cap finding, the important judgement is that it is the boundary of a Verified at head |
Two textual conflicts, both anticipated by the PR description, plus two semantic ones git could not flag. src/core/daemon/runtime.js, runMaintenance: #747 (LLP 0220) replaced the `withSpan` wrapper with a hand-rolled `tracer.startActiveSpan`, because the tick's clean/degraded status is only knowable once the report is in hand. Kept that skeleton, and put this branch's three statements back inside its `try` after `const report`: `summarizeMaintenanceSkips`, `persist`, and the `daemon.maintenance_skipped` line. The span attributes compose, with #747's `partitions_failed` / `partitions_maintained` beside this branch's `partitions_visited` / `partitions_skipped`, and #747's `setStatus` stays last so it still reads the finished report. llp/0218, the header: both sides appended an `**Extended-by:**` line. Merged to one line carrying LLP 0220 then LLP 0228, separated by `; `. LLP number collision: master landed 0224-desktop-setup-second-pass, so this branch's 0224-maintenance-skips-are-a-standing-surface was a second claimant on the same number and every `@ref LLP 0224#status-file-is-the-surface` would have been ambiguous. Renumbered the later claimant to 0228 per LLP 0156, with every inbound reference swept (llp/0217's forward-ref, the doc title, and the refs in src/core/daemon/{runtime,status,types.d}.js, src/core/commands/ status.js, and the test). Refs carrying 0224's own anchors (#repair-surface, #ask-once-per-pick) are the desktop doc's and were left alone. 0226 and 0227 are already claimed by open branches, so 0228 is the first free number above the highest claimed anywhere. MaintenanceReport.totalFailed: #747 made the field required, so this branch's test fixture no longer typechecked. Derived it from the partitions like every other total, and pinned the interaction it exposes with a new test: a partition carrying #747's `failed` is this tick's error, not a skip, so it stays off the skip surface and keeps its own per-partition `daemon.maintenance_failed` line (LLP 0220#this-tick-versus-a-recorded-one). npm test and npm run typecheck both produce a failure set byte-identical to an origin/master worktree in the same environment (23 test failures, 1 typecheck error, all pre-existing and environmental). The PR's own suite is 14/14. Smokes cache_lifecycle_maintenance, status_diagnostics, and daemon_foreground_start_stop are green. Co-Authored-By: Claude <noreply@anthropic.com>
One partition whose compaction throws no longer aborts the whole maintenance walk. Every partition behind it now gets its compaction and its snapshot expiry, and the tick reports the loss instead of propagating a bare exception.
The bug
maintainCachehad no per-partition catch andwithSpanrethrows, so a single throwing partition ended the walk. The daemon caught it, loggeddaemon.maintenance_failedwith no partition attribution, and hit the same wall an hour later.LLP 0199's neediest-first order makes it worse: the walk descends by file count, so the most fragmented partition goes first - exactly the one most likely to fail a rewrite, and exactly the one LLP 0217's thaw newly makes eligible. A failure there starves every other partition in the cache.
Where the catch went, and why
Around the whole
await withSpan('maintenance.partition', ...)call, outside the helper.withSpanrecords the exception and setsSpanStatusCode.ERRORbefore rethrowing, so catching outside keeps the trace honest about which partition failed while the walk continues. Catching inside the callback would publish anokspan for a partition that threw - the same legibility defect one layer down.Consequence: the partition report is now built in the loop body before the span opens and mutated in place, so the catch keeps what the run had already established (live file count, snapshots expired before compaction reached the error).
stampWriterGenerationis untouched. It fires fromcompactGeneration's own catch insidemaintainGeneration, before the error reaches the new one - so LLP 0217's "the attempt spends the retry" still holds. Pinned by test 2: the stamp andattemptFailedAtare present, the baseline is unmoved, nodataFilesBeforeis invented, and the next tick reports LLP 0218'scompactionAttemptFailedwithtotalFailed === 0.What the tick returns
It resolves, with the failures carried in the report:
failed/errorKind/errorMessageper partition, andtotalFailedon the report.Rejecting after the walk would have left the daemon wiring untouched, but it discards the report - so a tick that maintained 40 partitions and lost 1 would print nothing about any of them. Callers read the failures instead:
daemon.maintenance_failedper failed partition, now naming dataset, partition anderror_kind- which the propagated exception never could. Themaintenance.tickspan getsstatus: degradedandpartitions_failed. The outer.catchstays for what is still outside the per-partition catch (discovery, the retired-generation sweep).hyp query maintain: printsFAILED: <msg> (<kind>); the walk continued, aN partitions failedsummary, and exits 1.That exit code is a user-visible CLI contract change, deliberate: previously the exception propagated and the command exited non-zero, so a script gating on that would otherwise start reading a degraded run as clean. Documented in LLP 0220's consequences.
failedis not a reuse ofcompactionAttemptFailedThey mean different things and must not be conflated.
failedmeans this tick threw here; LLP 0218'scompactionAttemptFailedmeans an earlier tick's attempt failed and nothing has been attempted since. Test 4 asserts they never co-occur.Evidence
Test written first, driving the real
maintainCacheover a fixture with a throwing partition and a healthy one behind it in walk order: 8 identity-partitioned sessions with the stamp-less #723 cursor (so LLP 0217 owes a retry) and one live parquet file truncated to 4 bytes, pluslogs/source=claudewith 3 files.Against unmodified code, 4 of 4 fail, each because the exception escapes and the healthy partition is never reached. Independently re-derived by the reconciler:
After: 4/4.
Four
assert.rejects(maintainCache(...))calls incache-compaction-effectiveness.test.jsnow assertpartitions[0].failed === trueinstead - what they pin is unchanged, only how the failure surfaces. Both compaction smoke flows gained atotalFailed === 0assertion.Full suite 4005 pass / 0 fail / 1 pre-existing skip; typecheck clean;
llp-ref-hygiene11/11;cache-retention-maintenance38/38;cache_lifecycle_maintenanceandincremental_sink_compactionsmokes ok.LLP
New LLP 0220, with
Extended-byforward-refs on LLP 0199 and LLP 0218. Nothing 0199, 0217 or 0218 settled is contradicted - 0217 explicitly describes the no-catch abort as a defect consequence, and this documents the walk's survivability rather than changing when a partition is compacted. Number verified free across every remote branch (0219 is claimed by unmerged PR #745).Fixes #737